HDDS-13063. Support splitting json output to multiple valid json files in a directory#10604
HDDS-13063. Support splitting json output to multiple valid json files in a directory#10604prathmesh12-coder wants to merge 1 commit into
Conversation
sreejasahithi
left a comment
sreejasahithi
left a comment
There was a problem hiding this comment.
Thanks @prathmesh12-coder for the PR.
Left few comments.
| * Writes verification output to file(s) instead of stdout. | ||
| * When recordsPerFile is greater than zero, the keys are split into multiple valid JSON files. | ||
| */ | ||
| private void writeOutputToFiles(ObjectNode root, ArrayNode keysArray, boolean allKeysPassed) throws IOException { |
There was a problem hiding this comment.
writeOutputToFiles only writes new files, it never deletes existing ones. A re-run overwrites .0, .1, … but leaves higher numbered files from a prior run. This can be confusing to the user.
There was a problem hiding this comment.
When --out is used with --max-records-per-file, each split file only contains:
{ "keys": [ ... ] }
But stdout and single-file --out (without splitting) include a top-level "pass" field:
{ "pass": true/false, "keys": [ ... ] }
So the JSON shape is inconsistent across output modes. If someone parses the split files, they can’t see the overall pass/fail result unless they check each key’s "pass".
Not sure if this is intentional. If split files are meant to be key-only chunks, a short note in the help text would help. If all modes should match, consider adding top-level "pass" to split files too (or writing a small summary file).
There was a problem hiding this comment.
Tests cover split output (--out + --max-records-per-file) but not single-file --out alone. You could add a test for this case aswell.
3 participants
What changes were proposed in this pull request?
ozone debug replicas verify currently prints all verification results as one large JSON to stdout.
This PR adds support for writing output to a directory as multiple valid JSON files, similar to ozone debug ldb scan.
below options are added:
When set, output is split into files like dirName.0, dirName.1 etc. Each file is valid JSON with a keys array.
If --out is not used, the command still prints JSON to stdout as before.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13063
How was this patch tested?
Added integration tests in TestOzoneDebugReplicasVerify:
Verified split files are valid JSON, respect --max-records-per-file, and contain all keys.
Manually tested in local docker-compose Ozone cluster:
confirmed the directory is created, the output is split into numbered valid JSON files, and the run summary is printed.